Too much statistical power can lead to false conclusions: a response to 'Unsuitability of the epidemiological approach to bicycle transportation injuries and traffic engineering problems' by Kary.

Injury Prevention recently published a commentary critical of epidemiological approaches to cycling safety. The author, Kary, suggests that our paper, which reanalysed Walker’s study of motor vehicle overtaking distance for cyclists, made false claims about type I errors and confused statistical significance and clinical significance. Kary supports this critique, along with other points in the commentary, with a non-peer-reviewed response posted by him to the Journal’s website. In our paper, we note that, increasing power when computing sample size leads to an increase in the probability of a type I error. Kary incorrectly repeats this contention as the probability of a type I error increasing with sample size, suggesting ours was a ‘false claim’. Sample size when comparing two groups on a quantitative variable, such as comparing motor vehicle overtaking distance when wearing or not wearing helmet, is a function of effect size (δ), the type I error rate (α) and power. The sample size for a two-sample t test for a small effect size δ=0.2 with 80% power and α=0.05 is n=786 (or 393 per group). When power increases to 98%, as in the Walker study, the type I error rate increases to α=0.45. It is possible to maintain a nominal type I error while increasing power; however, the sample size more than doubles in this case (807 per group). Clinical significance relates to the practical implications of the absolute size of an effect. In the Walker study, the difference in motor vehicles overtaking helmeted and unhelmeted cyclists is at most 7 mm for manoeuvres <1.5 m. The largest absolute effect size is for overtaking at more than 2 m and amounts to 7.2 cm. There is no established appropriate effect size for overtaking distance, and neither Walker nor Kary has suggested one. However, there is unlikely to be an appreciable increase in safety when increasing overtaking distances from 2 to 2.07 m. Standardised indices, such as Cohen’s d, can provide a sample size independent quantification of effect size. Walker found a statistically significant relationship between helmet wearing and overtaking distance (F1,2313=8.71, p=0.003) which is equivalent to d=0.12. This is trivial by Cohen’s definition. As we aimed to demonstrate in our reanalysis of Walker’s data, both the calculated effect size and the absolute effect size do not support helmet wearing as a major factor in overtaking distance between cyclists and motor vehicles.